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Improving student performance, especially that of those 
learners who experience poverty, disabilities, language barriers, 
or some combination of these factors, is an ongoing challenge 
for many, if not all, school districts. It can be particularly 

so for smaller or more geographically isolated districts that, 
compared to their larger urban counterparts, are likely to have 
fewer resources at hand, including content specialization. 


California Education Partners (Ed Partners), a nonprofit 
organization that identifies itself as an education reform 
organization, recognized this issue when it began working 
with small California districts at its inception in 2011. By 
2015, it had come to see this resource gap as a critical issue 
for small- to medium-sized districts whose schools were 
struggling to improve instruction. In response, it launched a 
cross-district collaboration called the California Language & 
Learning Initiative (CALLI). Ed Partners intended CALLI to 
support such California districts both directly — on-site and 
at regular convenings — and indirectly by creating a structure 
through which participating districts could collaborate on 
their improvement efforts. 


Twenty-five districts signed on to start working with Ed 
Partners for three years starting in school year 2016/17 in one 
of three content areas: early literacy (13 districts), academic 
language (6 districts), and high school mathematics (6 
districts). Participants comprised elementary, high school, 
and unified districts, ranging from 4 to 30 schools and from 
1,122 to slightly more than 21,000 students. At the end of 

the first three years, districts were invited to continue for 
another two years in a modified approach called CALLI 201. 
Eighteen districts chose to do so. (For more information about 
CALLI’s approach, see Appendix A: About CALLI; for more 


About this Brief 


In spring 2017, WestEd was engaged 

to conduct a formative evaluation 

of the California Language & 

Learning Initiative — a cross-district 
collaboration developed and supported 
by California Education Partners (Ed 
Partners). Over the next four years, 
WestEd staff conducted a series 

of interviews to advise Ed Partners 

on participants’ perceptions of the 
initiative and their recommendations 
for improvement. During that period, 
WestEd conducted focus groups with 
district improvement team members 
(35 participants in five focus groups) 
and with team leads (49 participants 

in seven focus groups) as well as 
individual interviews with 25 improvement 
team members, 12 team leads, and 14 
superintendents from CALLI districts. 


This brief draws on the results of 
those interviews to present key 
learning about the initiative and what 
districts need to do to make the most 
of such collaborative opportunities. 
Note that WestEd’s evaluation was 
not a summative assessment of the 
effectiveness of Ed Partners’ approach 
to continuous improvement. 
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about participating districts, see Appendix B: 
Participating CALLI Districts). 


This brief draws from interviews with 
participating districts’ team members about what 
they and their districts gained and learned from 
the CALLI experience. 


What is CALLI’s value for districts? 


Members of district improvement teams engaged 
in CALLI identified a wide range of elements in 
the collaboration that created value for themselves 
and their districts, including direct support they 
received from Ed Partners, an opportunity to 
learn with and from peers, and support from 
content experts. They also praised the structure, 
accountability, and team time established by 


CALLI that helped them to develop and, as needed, 


refine a district improvement plan and to stay 
focused on the core work outlined in that plan. 
They reported seeing increased personal and 
team capacity to use continuous improvement 
processes. And they noted the development of 
an “improvement mindset” within their team, a 
mentality that values the iterative improvement 
cycles of planning, doing, and learning. 


Team members credited CALLI participation with 
helping them develop a better understanding of 
their district’s challenges and with developing 
greater trust and better communication among 
improvement team members and between team 
members and district leaders. They also highly 
valued becoming part of a content-specific CALLI 
community of learners with other districts. 

Each of the three content-specific communities 
was convened separately three times a year. 
Between convenings, community members were 
encouraged to reach out to one another for help. 
Interacting with the same participants over many 
convenings helped generate mutual understanding 


and trust, which, in turn, helped them more freely 
give and receive critiques of one another’s plans 
and processes, including suggested improvements. 


“If you care about student learning and 


student achievement and you want 


the freshness of ideas, you should 


[become] part of [a] collaborative. It’s 


worth the investment of time. It will 


pay itself back with student results 


and staff growth.” 


—District Superintendent 


Many participating districts were rural, and 

their team members noted that CALLI provided 
opportunities for professional growth and access 
to experts that would not otherwise have been 
possible. As one improvement team lead observed, 
“Without this, there is no way we would have 

the big-name [experts] — leading people in the 
discipline — [or the] technical support. There are 
just so many things that we would not have the 
money for.” 


CALLI participants noted advancement in 
several specific areas related to their district’s 
continuous improvement efforts. Some reported 
better district practices supporting continuous 
improvement cycles, such as routine discussions in 
school, district, and cabinet meetings of initiative 
evidence and its implications. Others reported 
developing new approaches for building broader 
buy-in for the district improvement plan. For 
example, members of one district team reported 
adopting an initial improvement strategy that 
had low stakes for teachers; the intent was to help 
teachers acclimate to the process and become 
more trusting of the plan before asking them to 
implement strategies with higher stakes. 
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Were there positive impacts on 
instruction and student achievement? 


In the early years of CALLI, districts used state 
test scores to find evidence of student progress. 
Some found it. One improvement team member 
recalled that after a departmentwide intervention 
in their district, student test scores “jumped 

so dramatically the state thought they had to 
re-evaluate the numbers and double-check that 
there wasn’t an error.” Most district teams, 
however, found that it takes time to develop 
improvement strategies, test and refine them, and 
implement them at the scale needed to generate 
widespread gains. 


With the onset of the COVID pandemic and 
limited statewide testing in California for the last 
two years of CALLI, districts could not easily track 
annual progress. Instead, team members relied 
on interim assessments and classroom assessment 
strategies, such as exit tickets (i.e., a short student 
response to a prompt about the lesson), to see if 
students showed improvement. They also relied 

to varying degrees on informal indicators to help 
them see if the classroom instructional changes 
they sought were happening. Several districts 
relied on teacher reports and anecdotal evidence 
of changed classroom practices and improved 
student learning. While team members reported 
that their data were positive, they acknowledged 
the need to improve the quality of the evidence 
they use to assess progress. 


Some districts found early evidence of changed 
practice in classroom observations. Many CALLI 
districts with high numbers of English Learners 
had identified academic language as their 
content focus for continuous improvement. Their 
improvement teams developed new instructional 
tools and techniques to increase students’ use of 
rigorous academic language. One district wanted 
to see a more engaged classroom. Its primary 
strategy encouraged teachers to pull students 


into discussions around mathematics even if it 
meant the classroom would be “a little messy or a 
little loud.” Over time, as teachers became more 
comfortable in urging students to “add a second 
sentence or to justify their thinking,” classrooms 
did indeed become louder. “It was good to have a 
noisy classroom,” the team member concluded, 
“because it meant kids were talking about math.” 


In another district, all teachers were trained to 
implement new strategies to engage students in 
vocabulary-rich academic discourse. They used 
empathy interviews to see what students thought of 
the new classroom practices. One improvement team 
member said, “Students crave these conversations. 
I have a student who typically does not initiate 
conversations. In an empathy interview, she told 
me she is willing to participate in our structured 
conversations because she knows the roles, and she 
looks forward to having these conversations.” 


What do districts think of Ed Partners’ 
approach? 


One theme articulated by CALLI participants over 
four years of formative interviews was a strong 
appreciation for Ed Partners’ overall approach to 
the work. Members of district improvement teams 
reported appreciating their relationship with the 
Ed Partners staff member assigned to work with 
their district. Many team members referred to 
their Ed Partners contact as an integral and vital 
part of their team. Some noted that their contact’s 
strong facilitation skills and broad perspective 

on California districts contributed to better team 
discussions and decisions. In addition to WestEd’s 
formative assessment of CALLI, Ed Partners 
regularly polled participants to see what they 
thought was or was not working in the initiative. 
Participants valued the opportunity to provide 
feedback. As one noted, “They’re always working 
to improve their processes as well.... And they are 
always learning, too. I appreciate that.” 
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“I would tell a fellow superintendent, 


‘You can't do this alone, and you 


shouldn't do this alone. If so, youre 


doing it the hard way.” 


—District Superintendent 


Perhaps because finding time to meet as a team 
is a constant challenge in the day-to-day frenzy 

of district work, many team members also said 

they valued the time for teamwork built into the 
schedule of the three annual convenings. 


Finally, teams praised the content-specific support 
they receive directly from Ed Partners and content 
experts brokered by Ed Partners. One team 
member said, “Ed Partners is not afraid to say, ‘It 
sounds like youre struggling...let’s give you some 
resources or bring in a [content expert].” 


How did districts use their CALLI 
training to address COVID? 


When COVID-19 forced the shutdown of California 
schools, districts encountered challenges they 

had not envisioned when initially developing 

their improvement plan. “Everything we've been 
working on is halted because we're no longer in 
front of students,” noted one participant. Districts 
were forced to quickly identify their most pressing 
COVID-related issues and develop a plan to 
address them. In interviews, participants said 
they relied on their improvement team and the 
protocols they had learned from Ed Partners (e.g., 
one district used empathy interviews to see why 
students were not logging into class) to create 

a new plan of action. “To deal with the COVID 
crisis,” said one interviewee, “we fell back on ... 
the way we collaborate [as a team], which had 
improved so much over the course of the year.... 
We're able to make great strides because...we 
already had in place our understanding of how 
our collaboration should work.” 


In one district, teachers struggled to teach 
remotely and engage students. The CALLI team 
realized they needed a simple solution to keep 
students reading complex texts. One teacher said, 
“With COVID, we thought, ‘what could we push 
while kids are distance learning?’ And one of those 
things was the volume of text that they are reading. 
We knew we had to plan something manageable. 
And we knew we had to plan something where it 
would be easy for us to get the data.” They decided 
to ask third graders to independently read two 
articles per week using the district’s online system 
that tailors texts to students’ reading levels. These 
students demonstrated greater growth in reading 
compared to a control group with whom this 
strategy was not used. 


In another district, by fall 2020, the improvement 
team saw an increase in the number of students 
receiving Ds and Fs on assignments. The 

team used empathy interviews and surveys to 
understand why students think they are earning 
poor grades. They learned that students wanted 
more interactive and engaging activities. Based on 
student responses, the team developed a learning 
cycle around activities that require students to 
interact and connect with peers through content- 
focused student-to-student conversation. 


What did districts learn about 
collaboration? 


Assemble the right team 


Many districts that decided to continue in CALLI 
201 rethought their improvement team based 

on their own experience and guidance from 

Ed Partners. For example, teams that had not 
included district-level leadership had struggled 
to gain broad support in the district. Similarly, 
teams without principals had struggled to gain 
support in schools, and those without teachers 
had struggled to get teacher buy-in. Teams found 
they needed representation from all three levels to 
develop a plan they could implement with broad 
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buy-in. “One reason our team was strong,” said 
one member, “is that we had teachers, principals, 
and district-level staff. If you don’t have that mix, 
you've got blind spots, and [for] any plans you put 
in place, you're going to miss a whole component.” 


Focus on culture 


Many team members who believed their team 

was having success attributed it to team members 
being willing to try solutions and potentially fail 
and to learn from failure. “We made a pledge to 
come in open-minded — to challenge ourselves to 
think differently about professional development and 
to develop something that would be sustainable.... 
We had no idea what [would be sustainable], but 
that was the mindset we wanted,” reported one 
superintendent. 


Many teams found it equally important to 
eliminate hierarchy within their CALLI team. 
Participants who engaged with other district 
teams at CALLI convenings noted that it was 
readily apparent when a team did not equally 
value all members’ voices. Such teams, they said, 
struggled to gain trust among their members and, 
thus, struggled to gain momentum. 


Across the board, district teams underscored 

the need to involve teachers in developing and 
refining their district’s improvement plan because 
implementation was inevitably more difficult 
without teacher buy-in. At an early convening, 
teams learned to use empathy interviews with 
teachers and other stakeholders to examine issues 
from different points of view. For some districts, 
this technique transformed their staff’s approach 
to understanding any problem. “Before, we had 
never even heard of empathy interviews,” said 

one team member. “Now, empathy interviews are 
always something that comes up as a potential way 
to gather data.” 


Choose the right-sized problem 


For many districts, using Plan-Do-Study-Act cycles 
to improve processes and practices was new. “We 
used to plan and plan and plan because we did not 
want to launch something and have it fail,” one 
team member recalled. “Now we launch lots of 
little trials and see what works and what doesn’t 
and learn from that.” 


CALLI participants appreciated the concept of 
what one interviewee called being “in the learning 
zone.” That meant exploring a problem with the 
intent of deeply understanding it and remaining 
open to potential solutions. Being in the learning 
zone helped teams develop a studied response to a 
problem rather than rush to judgment. For many 
teams, reminders that they were in the learning 
zone made it easier to seek and receive advice from 
thought partners, other districts, or stakeholders. 


Successful teams found they spent a lot of time 

in the learning zone before defining the problem 
they wanted to solve and the strategy they wanted 
to implement to address that problem. They also 
learned to narrowly define the first problem they 
wanted to solve. For example, one district wanted 
to raise test scores for 3rd graders but realized the 
problem they first needed to address was helping 
teachers develop better strategies to increase 
student conversation using academic language. 
Once the right-sized problem was identified, the 
team developed hypotheses and strategies for 
addressing the problem; implemented and tested 
each strategy; and, based on data from that test, 
decided whether to abandon it, refine it, or scale 

it as is. One participant noted the importance of 
not trying to do everything at once: “Let’s try one 
thing. Start small, then scale. That was the biggest 
learning for me.” 
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Identify the evidence needed to assess 
the work 


During their CALLI experience, teams increasingly 
focused on evidence: Did they have evidence about 
what was working? Could they measure the impact 
of improvement efforts for students and teachers? 
For many, the effort to find better evidence was 

a constant struggle. To begin with, they needed 

to learn what constitutes rigorous evidence. 

Many districts found that their evidence was not 
as rigorous and reliable as it needed to be. The 
collaboration helped teams develop better evidence 
through consultation with content experts, other 
district teams, and Ed Partners staff. 


Embrace problems and obstacles 


Participants reported that while their 
implementation efforts often failed, they had come 
to recognize failure as a learning opportunity 
rather than as the end of an improvement effort. 
Reflecting on the necessity of being “in the learning 
zone,” one participant said, “We are more reflective 
about failure. We analyze what went wrong and 
rethink how to execute based on that failure.” They 
also felt freer talking openly with other teams about 
what had not worked, thus allowing teams to learn 
from one another’s experiences. 


Think about sustainability 


Overall, CALLI teams identified participation 

in the collaboration as one of the best staff 
development activities they had ever undertaken. 
Team members could articulate what they had 
learned through the process and how it had 
changed their practices. Several team members 
reported that they had recently been promoted 
in their school or district to take on new 

roles in coaching, curricular leadership, and 
administration. One participant became a first- 
time superintendent. They said the professional 
growth they experienced through CALLI 
contributed to them being promoted. For some, 
that added responsibility meant that they could 
take the ideas they learned through CALLI and 
apply them to other content areas or grade levels. 
Others reported recognizing in their fellow team 
members qualities and strengths they had not 
known their colleagues have. 


When team members talked about how they could 
sustain the improvement work, those who saw that 
their team had representation from the classroom, 
school administration, and district leadership 
believed that effective practices would continue. 
One district leader said, “Our principals are the 
future leaders of the district. If we get them [to buy 
in] and be involved, these practices will continue.” 
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Appendix A: About CALLI 


CALLI districts were asked to make a three- 
year commitment. During that time, each 
six-to-eight-member district improvement team 
worked through CALLI to develop and refine 

a plan to improve classroom instruction in its 
identified content area. Teams analyzed data to 
identify problem areas, reflected on root causes, 
hypothesized potential solutions, and captured 
that work in an evolving improvement plan 
augmented by implementation and monitoring 
plans. Ed Partners asked teams to identify key 
enablers of and barriers to progress, assess 
what they wanted to change or keep, and adjust 
their plan accordingly. Sometimes the data they 
collected prompted teams to change the problem 
they wanted to address or the strategies they 
wanted to use. 


District teams with the same content focus 

were brought together in a learning community 
and convened three times a year for two days 

of professional learning and face-to-face 
collaboration (carried out virtually during the 
pandemic). During each two-day meeting, teams 
learned from content experts, engaged with new 
processes and tools, and worked within and across 
teams on issues related to their respective plans. 


A key Ed Partners goal for these gatherings was to 
develop a sense of mutual trust and support between 
participants that would extend not only between 
convenings but eventually beyond CALLI itself. 

To that end, Ed Partners had participants work 
collaboratively in interdistrict discussions to provide 
feedback on one another's improvement plans, 
discuss shared and unique challenges, and develop 
ideas for potential solutions. Ed Partners encouraged 
teams to reflect individually and with other teams 
from their community of learners to diagnose what 
was working, what was not, and what might need to 
change in their plan. This collaborative inquiry was 
at the heart of the improvement process, allowing 


teams to learn from one another and to adopt or 
adapt practices and tools that had been successfully 
used in other districts. 


Finally, each district was assigned a dedicated Ed 
Partners professional to support the leader of its 
improvement team, serving as a thought partner 
and advisor between convenings. These individuals 
also brought in content area experts as needed. 
They often pointed to other districts facing similar 
challenges and encouraged team members to 
contact their counterparts in those districts. 


The emergence of CALLI 201 


During CALLI’s first three years, Ed Partners 
routinely assessed it using participant surveys, 
formative evaluation reports, and staff reflections. 
Based on the results of those assessments, Ed 
Partners made changes intended to strengthen the 
initiative and invited the original CALLI districts 
to extend their commitment by another two 

years (2019/20 and 2020/21) by signing on for the 
revamped program called CALLI 201. Eighteen of 
the original 25 districts accepted the invitation. 
Two of the 18 districts decided to participate in 
two content areas. (See Appendix B for more about 
participating districts.) 


CALLI 201 reflected three significant changes. 

Ed Partners brought on new staff who had 

more school and district experience than their 
predecessors and could thus offer better guidance 
to district teams. It also developed a two-year plan 
with a tighter focus on continuous improvement, 
using rapid cycles of testing and learning. Finally, 
it asked districts to (a) rethink the composition 

of their improvement team to ensure that the 
team appropriately represented the range of 
stakeholders needed for success and (b) establish a 
district leadership team to support and elevate the 
work of the improvement team. 


In CALLI 201’s first year, Ed Partners introduced 
its continuous improvement framework. Teams 
used a series of guiding questions to define a 
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challenge area they wanted to tackle and set a 
district goal for results the team wanted to change. 
The teams then engaged in a series of activities to 
analyze past efforts, current data, and the system 
that produced the results the team wanted to 
change. Some teams opted to continue working 
on the same challenge they had addressed in 

the first three years of CALLI. Others decided 
they needed to address a new challenge. These 
teams then developed a new improvement plan, 
with hypotheses linking team actions to desired 
outcomes and identifying the evidence they 
needed to monitor results. Finally, they developed 
an implementation plan specifying who was 
responsible, key activities, and a timetable. 


Teams committed to conducting a series of tests 
using a Plan-Do-Study-Act (PDSA) protocol to 
test their hypotheses, monitor results, and learn 
what worked and what did not. Ed Partners 
encouraged them to undertake rapid PDSA cycles 


to test small changes in practice and assess their 
effects on desired outcomes before expanding the 
solution. District teams identified stakeholders 
whose buy-in would be critical for success and 
developed strategies to increase their support for 
the improvement plan. They also defined the data 
and evidence they would monitor to understand 
the effectiveness of their work. 


Again, this was an iterative process, with teams 
conducting small tests; gathering evidence from 
stakeholders about the strategy being implemented; 
and, based on that data, deciding whether to refine, 
abandon, or scale the strategy. Between convenings, 
each district team continued implementation of 

its improvement plan and assessed its progress. 
During these periods between convenings, districts 
continued to receive expert support from their 
assigned Ed Partners staff member and, as needed, 
from outside content experts. 
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Appendix B: Participating CALLI Districts 


Free or 
Reduced- | English | CALLI 
Price Learners PAT 
District Enrollment | Meals (%) | (%) Districts 
Azusa Unified 19 9,277 87.4 33.0 x 
- Corning Union Elementary 6 2,043 83.1 34.4 
2 Cutler-Orosi Joint Unified 8 4,083 95.5 48.5 x 
g Dinuba Unified 10 6,580 79.5 22) e 
- Fairfield-Suisun Unified 30 21,366 59.4 15.3 . 
e Fowler Unified 9 2,477 75.8 22.6 
: Galt Joint Union Elementary 8 3,693 67.2 21.0 x 
o 
3 Galt Joint Union High 4 4,526 237 ae x 
5 Golden Plains Unified 6 1,831 91.2 62.0 S 
E Kings Canyon Joint Unified Zi 9,775 80.3 33.4 
: Lawndale Elementary 10 6,300 78.2 32.8 
: Mendota Unified qi 3,146 95.0 68.7 Ne 
§ Novato Unified i 8,029 35.4 18.0 x 
: Pixley Union Elementary 2 1,122 94.7 71.8 
€ Placer Union High Z 4,137 278 7 x 
i Red Bluff Union Elementary 7 2,163 71.8 14.3 
E Robla Elementary 6 2,231 90.6 42.3 x 
: San Rafael City Elementary 10 4,635 62.1 43.4 
é San Rafael City High 4 2,365 42.5 14.5 x 
: Sanger Unified 20 11,204 [33 20.2 x 
: Santa Paula Unified 10 5,459 84.1 41.5 S 
i 
i 
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Free or 
Reduced- | English CALLI 
No. of ligu (us Learners | 201 

District Schools Enrollment | Meals (%) | (%) Districts 
Selma Unified 12 6,447 78.6 29.0 x 
Tahoe-Truckee Unified 12 3,978 44.2 DDD x 
Ukiah Unified 15 6,349 TZ 26.1 x 
Washington Unified 23 7,978 63.9 20.1 x 


Source: California Department of Education 2014/15. 
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